Chapter 15 File Operations in Python

Note the following:-

  1. This html document is meant as an accompaniment to Chapter 15 File Operations in Python.
  2. The document contains scripts executed on IDLE as well as on Jupyter notebook.
  3. The scripts executed on Jupyter can be directly copied and run into a Jupyter notebook or some other IDE (Like Pycharm or Eclipse with PyDev or Visual studio).
  4. However the scripts on IDLE also contain the >>> symbol and therefore cannot be directly executed. If you want to execute them on IDLE or Jupyter, you need to manually remove the >>> symbol.
  5. Wherever needed some background material from the book is also included to help you better understand the scripts
  6. The topic numbers given on each paragraph match the topic numbers of the book, so you can easily identify the topics and corresponding scripts.
  7. In some of the scripts, the file paths give are that of the author's computer. You need to replace them with file paths of your own computer.
  8. At some places, to improve readability, page numbers of the book are indicated in green font like:- See Page 181 of the book
  9. This document was first created as a Jupyter Notebook as combination of Markdown and code cells (extension .ipynb) and then downloaded as html. If someone wants to "modify" or "extend' this document, you may ask for the original .ipynb file by sending me an e-mail at:- 999.anuraggupta@gmail.com

15.2. Basics of file operations in Python
15.2.3. The file_name argument in open() function

As indicated earlier, the file_name parameter must be given. It is the path to the file. The path can be relative or absolute. The relative path is from the current working directory (cwd). One can get the current working directory by using the getcwd() function of the os module as follows:
This script is available on page 369 of the book

>>>import os            #The os module has to be imported
>>> os.getcwd()            # This function of os module will give cwd
'C:\\ag\\myScripts'# The extra backslash is escape sequence
>>>print(os.getcwd())     # The print statement will remove the extra backslash
C:\ag\myScripts
>>> os.listdir()             # Will list all the files in cwd
['classInheritenceExample.py', 'dog_test.py', 'multipleInheritenceDemo.py', 'test_test.py', 'we_the_people.txt']
>>> os.chdir('C:\python_scripts')    # use chdir() to change directory 
>>> os.getcwd()            # output confirms directory changed to C:\Python
'C:\\python_scripts'

The file_name parameter can be the absolute path name or the relative path name of the file. For example on the author’s system, the absolute name will be: C:\ag\myScripts\we_the_people.txt and the relative path name will be simply we_the_people.txt. The following script creates two file handles on this file using absolute path and relative path. It also checks the type() of the file handle you got.

>>> fHandle = open("we_the_people.txt")       #Relative path
>>>print(fHandle)
<_io.TextIOWrapper name='we_the_people.txt' mode='r' encoding='cp1252'>
>>> fHandle2 = open("C:\ag\myScripts\we_the_people.txt") #Absolute path
>>>print(fHandle2)
<_io.TextIOWrapper name='C:\\ag\\myScripts\\we_the_people.txt' mode='r' encoding='cp1252'>
>>>

15.2.4. Using open() function to create a new file

Note: If you open a file with a file name which does not exist, Python will create the file for you. This can be understood from the following code:

>>> fHandle = open("we_the_people.txt")
>>> fHandle2 = open("C:\Python34\myScripts\we_the_people.txt")
>>> os.listdir()
['we_the_people.txt']
>>> fHandle3 = open("newFile.txt", "w+") # Creates a new file if it does not exist
>>> os.listdir()#Output confirms new file created
['newFile.txt', 'we_the_people.txt']
>>> fHandle4 = open("newFile2.txt", "a")

15.2.5. file object, access modes

The following example show how the file object (or file handle) attributes are used
This script is available on page 371 of the book

>>> fHandle = open('C:\Python34\myScripts\we_the_people.txt', 'r+')
>>> fHandle.closed  # Since file is open, will return False
False
>>> fHandle.mode  # The mode is same as given while opening the file
'r+'
>>> fHandle.name # Double back slash in output because \ is escape sequence in Python
'C:\\Python34\\myScripts\\we_the_people.txt'
>>>print(fHandle)  # Can also get file info by using print on the file handle
<_io.TextIOWrapper name='C:\\Python34\\myScripts\\we_the_people.txt' mode='r+' encoding='cp1252'>
>>>

To explain file operations, create a new file called derozio.txt with the following text:

My country! In thy days of glory past A beauteous halo circled round thy brow and worshipped as a deity thou wast—

Let us use this file derozio.txt to explain these file operations:
This script is available on page 373 of the book

>>>import os
>>> os.chdir("C:\Python34\myScripts")
>>> fObj = open('derozio.txt')
>>> fObj.read(10) #Reads first 10 bytes
'My country'
>>> fObj.tell() # File cursor is at 10
10
>>> fObj.read() # Reads rest of file from cursor position
'! In thy days of glory past\nA beauteous halo circled round thy brow\nand worshipped as a deity thou wast—'
>>>

15.3.2. seek(offset[,from_what]) method

As pointed out earlier, a “file object” has a method fobj.tell() which gives the “current position” of the “cursor” or “pointer”. The fobj, i.e., the Python file object also provides a method seek() to “change” the location of the cursor to a desired location. Some important aspects of the seek(offset[, from]) method are as follows:

  • It has two parameters: “offset” and “from_what”. The “offset” parameter is mandatory, while the “from_what” is optional.
  • The “offset” parameter indicates the “number of bytes” by which the cursor is to be moved.
  • The “from_what” parameter is the “anchor” or the point from where the count of “offset” begins. By default from_what is 0 meaning that by default the count of “offset” is done from “start” of the opened file. Similarly a value of 1 indicates that the “offset” is to be done from the “current position of the cursor”. A value of 2 indicates that the count of “offset” is to be done from the “end” of the file. These three values are also stored in following constants of the os module: os.SEEK_SET is equivalent to the value of from_what = 0; os.SEEK_CUR is equivalent to from_what = 1; and os.SEEK_END is equivalent to from_what = 2.

This is clear from following the following code:
This script is available on page 374 of the book

>>> fObj = open('derozio.txt')
>>> fObj.tell()
0
>>> fObj.seek(0,2) # The second argument value 2 moves to end of file
116
>>> fObj.tell()# Confirm that cursor at  end of file (File has 116 characters)
116
>>> fObj.seek(0,0) # The second argument 0 moves cursor to beginning of file
0
>>> fObj.seek(10) # Moves cursor to 10th character
10
>>> fObj.read() # Will read from 10th character onwards
'! In thy days of glory past\nA beauteous halo circled round thy brow\nand worshipped as a deity thou wast—'
>>>

15.3.3. readline([size]) and readlines([sizehint]) (note: the singular and plural)

The syntax for the readlines() is

fileObject.readlines( [sizehint] );# Square brackets indicate optional parameter

The following code example on IDLE (using the previously created file derozio.txt) explains the concepts and usage:
This script is available on page 374 of the book

>>> fObj = open("derozio.txt")
>>> myStr = fObj.readline()
>>> myStr
'My country! In thy days of glory past\n'
>>> type(myStr)
<class'str'>
>>> myL = fObj.readlines()
>>> myL
['A beauteous halo circled round thy brow\n', 'and worshipped as a deity thou wast—']
>>> type(myL)
<class'list'>
>>>

We can read a file line by line using a while loop as follows:
This script is available on page 375 of the book

In [1]:
fObj = open('C:\Python34\myScripts\derozio.txt', 'r+')
currL = fObj.readline() # currL is a list of lines in the file
while currL:
    print(currL)
    currL = fObj.readline()
fObj.close() # Should close the file

My country! In thy days of glory past

A beauteous halo circled round thy brow

and worshipped as a deity thou wast—

15.3.4. Writing to files using write() method

The following code shows how the open() function creates a file which did not exist before. It also shows use of read(), write() and seek() methods.
This script is available on page 376 of the book

>>>import os
>>> os.listdir()
['derozio.txt', 'newFile.txt', 'newFile2.txt', 'we_the_people.txt']
>>> f = open(r'C:\Python34\myScripts\test.txt','w+')
>>> os.listdir()
['derozio.txt', 'newFile.txt', 'newFile2.txt', 'test.txt', 'we_the_people.txt']
>>> f.read()
''
>>> f.write("We shall overcome")
17
>>> f.seek(0,0)
0
>>> f.read()
'We shall overcome'
>>>

15.3.5. writelines(aList) method of file object A file object also has a writelines() method. The method writelines() writes a “sequence” of strings to the opened file. The sequence can be in form of an “iterable” object like a “list of strings”. The method writelines() also has a return value of None. The syntax is as follows:

fileObject.writelines( someSequence )

Where someSequence is typically a list. Note that newlines are not added. The following code will explain the concept:
This script is available on page 377 of the book

In [2]:
myL = ['A quick', 'brown fox', 'jumped', 'over', 'a lazy dog']
with open('test.txt', 'w') as f:
    f.writelines(myL)
with open('test.txt', 'r+') as f:
    print(f.read())

A quickbrown foxjumpedovera lazy dog

15.4.1. Using the “with” statement to close file automatically

This method of using the “with” statement to open a file in a block of code is shown in the following example code:
This script is available on page 378 of the book

In [3]:
path = 'C:\Python34\myScripts\derozio.txt'
with open(path, 'r+') as fObj:
    for line in fObj:
        print('*',line)

* My country! In thy days of glory past

* A beauteous halo circled round thy brow

* and worshipped as a deity thou wast—

15.4.3. Some scripts in Python to demonstrate concepts of file usage

Code for deleting a particular line number in a file:

In the following script, there is a function fDelLine() which takes two parameters. The first parameter is the file name from which a particular line has to be deleted. The second parameter is the line number which has to be deleted. The script uses a file named temp.txt which has the following three lines of text:

My country! In thy days of glory past A beauteous halo circled round thy brow and worshipped as a deity thou wast—

The following script will delete Line number 3 ( Note that when one says 3, it means starting from 1 and going upto 3. But in a list, the index starts from 0, so the start from 1 but in the list object that you store these lines, the index starts from 0)
This script is available on page 378 of the book

In [4]:
def dLine(myF, lineNumb):
    with open(myF, 'r+') as f1:
        myL = f1.readlines()
        if lineNumb > len(myL):
            print("Cannot delete since lines in file ->", len(myL))
        else:
            dLine = myL.pop(lineNumb-1)
            print('Deleted line:-', dLine)
    with open(myF, 'w+') as f2:
        f2.writelines(myL)

dLine(r'C:\Python34\myScripts\test.txt',1)  

Cannot delete since lines in file -> 0

Code for creating copy of a file:

The steps are:

  1. Open the source file for reading from it.
  2. Open a new file for writing purpose.
  3. Read data from the first file (can be string at a time or complete data of file.)
  4. Write the data read in the new file.
  5. Close both the files.

Following is a script for a function which takes two parameters. The first parameter is the source file, i.e., file to be copied. The second file is the destination file, i.e., the file to which data from first file is copied.
This script is available on page 379 of the book

In [5]:
def fCopy(fOrig, fCopy):
    with open(fOrig, 'r+') as f1:
        myS = f1.read()
        with open(fCopy, 'w+') as f2:
            f2.write(myS)
# Use the function
fCopy(r"C:\PythonScripts\derozio.txt", r"C:\PythonScripts\copy_derozio.txt")

Program code for deletion of the line(s) having word (passed as argument)

Here you have written a script which consists of a function which takes three parameters.

  • The first parameter is the source file name, i.e., the file to be read.
  • The second parameter is the destination file name, i.e., the file to which all lines not containing a particular string are written.
  • The third parameter is a string which if present in a line will cause that line not to be written to the destination file.

For this script, a file called cat.txt was created. Contents of this file are as follows:

caterpillar dog catapult lion catholic tiger catalyst mouse

Note that every alternate word “contains” string “cat”. The script creates a new file called nocat.txt which has only those lines which do not have the string “cat”. The script is as follows:
This script is available on page 380 of the book

In [6]:
def delWord(myF1, myF2, myWord):
    f1 = open(myF1, 'r+')
    f2 = open(myF2, 'a')
    for line in f1:
        if myWord not in line:
            f2.write(line)
            print(line)
# Function call
delWord(r"C:\PythonScripts\cat.txt", r"C:\PythonScripts\nocat.txt","cat")

dog

lion

tiger

mouse

15.4.5. Pickle module and steps in unpickling

Steps in unpickling are

  • Import pickle module.
  • Open the file and create a handle to the file or a file object from which to unpickle. Open the file in 'rb' mode, i.e., “for reading in binary mode”.
  • Use the method pickle.load(input_file). This method of the pickle module reads the saved data in the form of a stream. Thereafter, it “re-constructs” the object from the data stream.

The following code shows how pickling and unpickling of a dictionary is done:
This script is available on page 382 of the book

>>> fileObj = open('C:\Python34\myScripts\myPick.pkl', 'wb')
>>>import pickle
>>> myDict = {'a':1, 'b':2}
>>> pickle.dump(myDict, fileObj)
>>> fileObj.close()
>>> f = open('C:\Python34\myScripts\myPick.pkl', 'rb')
>>> myObj = pickle.load(f)
>>> myObj
{'a': 1, 'b': 2}
>>> f.seek(0,0)
0
>>> f.read()
b'\x80\x03}q\x00(X\x01\x00\x00\x00aq\x01K\x01X\x01\x00\x00\x00bq\x02K\x02u.'
>>>

15.6. Writing small scripts for inserting data in a file

As pointed out earlier there are two possibilities when writing to a file. The first is in the write mode when the original contents of the file are simply overwritten. The other possibility is when new content is added to the file. Now the append mode of the write method will append, i.e., add the data to the end of the file no matter where the file pointer is. This means that in Python there is no method to add data to the beginning or to middle of a file. If you want to add data to the middle of a file, then you have to do this by writing your own code. There are at least two ways to append data in the middle of a file:

First method: The steps in the algorithm are as follows:

  1. Get the data to be inserted.
  2. Open the original file say f1 in reading mode.
  3. Open or create another file say fTemp for temporary storage.
  4. Read the original file sequentially and keep on writing the data to the temporary file till you reach the point where new data has to be inserted.
  5. Write the new data to the temporary file.
  6. After having written the new data continue with step 4 above till the EOF is reached.
  7. Delete the original file or overwrite the original file with the temporary file.

Following is a script which modifies a file original.txt. The contents of this file are as follows:

11111111 22222222 33333333 44444444 55555555

The file has 5 lines of text consisting of numbers. The following script adds a new line of text consisting of 0000000 to the file at location 2nd line and stores it in a new file called modified.txt.

The script has a while loop with an if-else block. The if block is meant to add the string myS to the temporary file while the else block is meant to add the lines before and after the line number to be added to the temporary file.

In [7]:
def add_line(myF1, myF2, myS, numb):
    f1 = open(myF1, 'r+')
    f2 = open(myF2, 'a')
    count = len(f1.readlines())
    f1.seek(0,0)
    n = 0
    flag = True
    while n < count:
        if((numb -1) == n) and (flag == True):
            f2.write(myS)
            print('Added text', myS)
            flag = False
        else:
            myL = f1.readline()
            f2.write(myL)
            print('Added line', n, myL)
            n = n+1
# Test the script
myF1 = r'C:\testData\original.txt'
myF2 = r'C:\testData\modified.txt'
myS = r'0000000'
add_line(myF1, myF2, myS,2)

Added line 0 11111111

Added text 0000000
Added line 1 22222222

Added line 2 33333333

Added line 3 44444444

Added line 4 55555555

Second method: The steps in the algorithm are as follows:

  1. Create a file object of the source file. Read the source file into a list using the readlines() method of file object.
  2. Manipulate the list returned by the readlines() method. You can add, delete or modify the individual items of this list.
  3. Create a file object of the destination file. Write the manipulated list back to the destination file object using the writelines() method of the file object and giving it the manipulated list as a parameter.

This is clear from the following example:

In [8]:
def add_line2(myF1, myF2, myS, numb):
    f1 = open(myF1, 'r+')
    f2 = open(myF2, 'a')
    myL = f1.readlines()
    if numb <= len(myL):
        myL.insert(numb-1, myS + '\n')
        f2.writelines(myL)
    else:
        myL.append(myS + '\n')
        f2.writelines(myL)
    f1.close()
    f2.close()   
# Function call
myF1 = r'C:\testData\original.txt'
myF2 = r'C:\testData\modified.txt'
myS = r'0000000'
add_line2(myF1, myF2, myS,2)

15.9. Beyond text book d. Context manager in Python:
See Page 388 of the book
In the chapter it was explained that there is a way by which Python automatically closes opened file. The syntax was as follows:

But how does this automatic opening and closing of files happen?

The answer to this is a concept called “Context manager”.

Before discussing “Context manager”, one needs to understand that an important issue in programming is “Resource management”. Any program obviously will use resources. A “resource” is some component used in computing which has limited availability.

Some example of use of resources are:

  • When you open a file and load it in memory, then it is using up the RAM of your computer.
  • In a multi-threaded program, you may lock up a resource for one thread, thereby denying it to other programs.
  • A socket

Context manager: A context manager is a mechanism which automates the process of resource acquisition. Thus, a context manager prevents “resource leakage”.

The term “resource leakage” basically means not “releasing” a resource even after an application has finished using it.

For example, suppose a program opens a files and loads it in memory but forgets to close it after using it. This would lead to “resource leakage”.

Similarly suppose a program “locks” a file for editing so that other programs cannot use it, but crashes before it can release the lock. This also would lead to “resource leakage”.

You can make a Python class a “context manager”. To do this, let us first understand how the context manager works. The way Python implements context manager is as follows:

  • If the class implements a __init__(), method, then this method is called first.
  • For a class to act as a context manager, it must implement the following two methods:
  • An__enter__() method
  • An __exit__() method
  • So when you enter the context, using the “with” keyword, then first the __init__() method (If it is present) is called and thereafter, the __enter__() method is called.
  • When you leave the context, the __exit__() method is called.

The signature of the __exit__() method is:

object.__exit__(self, exc_type, exc_value, traceback)
  • Note that if an error is raised in __init__() or __enter__() , then the code block inside the “with” context, is never executed and __exit__() is not called.
  • However, if the __init__() and __enter__() methods do not cause an exception, then the code block will always be called.
  • The __exit__() method causes exit from the context. This method also returns a boolean flag. If an exception occurred during execution of the body of the “with” statement, then the arguments contain the exception type (exec_type), value (exec_value) and traceback information. (Note that the __exit__() method must contain three arguments to hold: (1) Exception type, (2) Exception Value and (3) Traceback information.
  • If no exception occurred in executing the block of the “with” statement, then all these three arguments, (1) Exception type (2) Exception Value and (3) Traceback information, are null.
  • Further, the __exit__() method returns a bool value. If the bool value returned is False, then the “with” block will not suppress the exception.
  • However, if the __exit__() method returns bool True, then this method will cause the with statement to suppress the exception and continue execution with the statement immediately following the with statement.
  • All these three methods (if __init__() exists) or two methods (if __init__() is not there) are called automatically by the Python interpreter. You don’t need to call them, but you need to implement them in your class for your class to work as context manager.
  • The advantage of making a class a “context manager” is that you don’t have to call open() and close() repeatedly.

The following example will clarify the concept:
This script is available on page 389 of the book

In [9]:
class MyContextClass():
    def __init__(self):  # To implement context __init__() is optional
        print('calling-> __init__()')
    def __enter__(self):  # To implement context __enter__() is compulsory
        print('calling-> __enter__()')
        return self
    def __exit__(self, exc_type, exc_val, exc_tb):
        print('calling-> __exit__()')

with MyContextClass() as cc:
    print(type(cc))  # cc is object of type MyContextClass()
    print('inside context of MyContextClass')

calling-> __init__()
calling-> __enter__()
<class '__main__.MyContextClass'>
inside context of MyContextClass
calling-> __exit__()